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ABSTRACT 
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educational researchers and educational practitioners has seen a great 
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education research evidence and examines the nature and quality of current 
educational research on students and student learning. There are three focal 
points in this review: (1) the increasingly diverse nature of the research 

field in environmental education; (2) the need for reviews focused on the 
nature of the research; and (3) recognition of the importance of research on 
students and learning. The methodologies used in this review are systematic, 
comprehensive, and analytical. The materials included in the review involve 
any environmental education activity undertaken in school or under the 
auspices of the school. The Review Framework is appended. (Contains 48 
references . ) (YDS ) 
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INTRODUCTION 

Educational research in the UK has recently seen a growth of interest in reviews of 
research as a means of improving the links between educational researchers and 
educational practitioners (Foster and Hammersley, 1998; Bassey, 2000; Davies, 
2000b; Evans and Benefield, 2000). This comes in response to considerable criticism 
over recent years about the limited impact of educational research on policy and 
practice, and the development of notions of evidence-based and evidence-informed 
practice in education (e.g. Hargreaves, 1996; Tooley and Darby, 1998; Hillage et al., 
1998; Davies et al., 2000). 

Looking within the field of environmental education research, there are signs of 
similar kinds of arguments and developments. Palmer (1999:394), for example, 
proposes a need for ‘policy and practice in environmental education to continue their 
movement away from a randomness of approach that leaves much to chance, towards 
research-based endeavours that encapsulate reliability and some of the realities of 
human motivation and cognition’. In a similar way, Reid’s (2001 :4) introduction to a 
recent edition of a practitioner journal argued that: ‘Inspecting the evidence base is a 
welcome corrective to authoritarian exhortations and political conjuring that tries to 
make some forms of environmental education appear while others disappear’. His 
view is that ‘good and bad practice in environmental education should rely on 
tradition and the power of persuasion less, and on scholarship and enquiry more’ 
(ibid.:4). 

Against the backdrop of these kinds of arguments, this paper reflects upon the 

experience of undertaking a review of research evidence in a particular part of the 

environmental education research field - namely, recent empirical studies of learners 

and learning in primary or secondary school environmental education. The purpose 

of the review was to examine the nature and quality of the current research evidence 

on students and their learning in environmental education. The concern with evidence 

was motivated by two considerations. Firstly, it responded to the tendency of 

previous analyses of the field to focus on methodological trends more than research 

findings (e.g. Williams, 1996; Palmer, 1998; Hart and Nolan, 1999). Secondly, it 

recognised that claims have been made that environmental education theory and 

research have overlooked ‘the children who are the subjects of environmental 

education’ (Payne, 1998:20). The review contended that efforts to address such 
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shortcomings need to be informed by a thorough and grounded understanding of what 
studies have, and have not, been undertaken on students and learning, and what is 
known, and not known, from the evidence that these studies have generated. 

The overall report and findings of the review will be published later this year 
(Rickinson, forthcoming). 1 The purpose of this paper, therefore, is not to discuss 
what the review found out about the evidence base on learners and learning. Rather, 
it is to problematise the review’s methodology, through an exploration of three 
aspects of the review process that were either problematic during its undertaking or 
have become the source of concern since the project’s completion. These focus on 
the processes of critiquing research evidence; synthesising research findings; and 
suggesting implications for research users. The underlying aim of this paper is to 
explore further the possibilities, and the challenges, of trying to make sense of 
research evidence in environmental education. 

The paper will begin with an outline of the review’s aims and rationale, followed by a 
description of its methods and approach. Attention will then turn to the three aspects 
of the review process that were experienced as problematic. A brief consideration of 
emerging issues in relation to reviewing environmental education research evidence 
concludes the paper. 

REVIEW AIMS AND RATIONALE 

The review focused on a particular part of the environmental education research field; 
namely, studies published between 1993 and 1999 that had investigated, in some way, 
learners and learning in the context of primary or secondary school environmental 
education. The aims of the review were threefold: 

♦ To chart the nature of the current evidence base on learners and learning in 
environmental education in terms of its major foci, strengths and weaknesses. 

♦ To identify key messages emerging from this evidence base for each of the major 
foci within the area, and assess the limitations of these in terms of empirical 
underpinnings and generalisability. 

♦ To raise questions about the nature, quality and accessibility of recent 
environmental education learning research, and suggest priorities for future work. 




1 A summarised version of the review’s main outcomes was also published recently in a UK practitioner 
journal (Rickinson, 2001). 
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The rationale for this undertaking encompassed several considerations. Firstly, recent 
descriptions of the field have highlighted its rapidly expanding size and increasingly 
diverse nature, particularly over the last ten years (Palmer, 1998; Hart and Nolan, 
1999). In view of this growth and diversification, a need was seen for work focused 
on making sense of and critically analysing the field of environmental education 
research. 

Secondly, the review argued that there was a particular need for reviews focused 
specifically on the nature and quality of the empirical evidence within the field. This 
recognised the fact that several of the reviews that have been undertaken in the field 
have been either bibliographic in nature (Iozzi, 1981; Andrew and Malone, 1995; 
Marcinkowski and Mrazek, 1996; Wagner, 1997; Foskett and Marsden, 1998), or 
methodological in focus (Robottom and Hart, 1993; Williams, 1996; Palmer, 1998). 
Furthermore, those reviews that have been concerned to collate and analyse research 
findings have tended to do so for particular kinds of studies on very specific topics. 
Examples include reviews of research on: the affective domain (Iozzi, 1989); 
knowledge and affect (Zimmermann, 1996), knowledge and behaviour (Newhouse, 
1990), interventions and behaviour (Zelezny, 1999), learning outcomes (Leeming et 
al., 1993), outdoor experiences and attitudes (Keighley, 1997), sources of 
environmental sensitivity (Chawla, 1998b), and student understanding of global 
atmospheric issues (Boyes and Stanisstreet, 1996). These articles have provided 
useful overviews of recent research findings, often with detailed critical commentary 
and discussion of implications for practice (e.g. Iozzi, 1989). Their specificity of 
focus, however, has usually meant that they have discussed studies of a broadly 
similar methodological nature. (An important exception is Chawla’ s (1998b) review 
of research on sources of environmental sensitivity which discussed a variety of 
qualitative and quantitative studies). 

Therefore, the review sought to focus specifically on the nature of the research 
evidence in an area of the field that was broad enough to include studies with a variety 
of approaches and foci, but also focused enough to enable detailed examination and 
discussion of individual studies’ findings. Research on learners and learning was seen 
as an area that fulfilled these requirements in that it represented a definable area 
within the literature that contained a variety of research topics and approaches. 

Thirdly, the focus on learners and learning also responded to growing recognition of 
the importance of research on students and their learning, both within the field of 
environmental education research (Payne, 1998; Rickinson, 1999), as well as in 
educational research debates more generally (Erickson and Shultz, 1992; Cooper and 
McIntyre, 1996; Rudduck et al., 1996; Pollard et al., 1997; McCallum et al., 2000). 
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Arguments made by such authors suggest ‘a lack of consideration in environmental 
education theory and research practices about the children who Eire the subjects of 
environmental education’ (Payne, 1998:20), and a situation in educational research 
more generally where ‘virtually no research has been done that places student 
experience at the centre of attention’ (Erickson and Shultz, 1992:467). The review 
aimed to establish insights into what studies had, and had not, been undertaken on 
students and learning, and what was known and not known from the evidence that 
these studies had generated. 

Fourthly, the concern with research findings drew upon the notion of evidence- 
focused reviews, in particular Foster and Hammersley’s (1998:610) argument that 
‘the main channel of communication between researchers and lay people ought to be 
reviews of whole fields of research, rather than reports of single studies’. This was 
not to suggest an unproblematic acceptance of the concept of evidence-based practice 
or a simplistic view of the relationship between research and practice. Rather it was 
to argue for a concern for research evidence and what this might tell us in the field of 
environmental education. 

Finally, this review sought to complement Hart and Nolan’s (1999) recent analysis of 
the field. In adopting a similar time-scale for included studies, it aimed to examine the 
field at a similar time in its development. However, in specifically focusing upon a 
particular part of the field, it was hoped that this review would be able to explore the 
findings of individual studies in more detail than was possible in Hart and Nolan’s 
wide-ranging analysis of the entire field. Indeed, the delineation of the scope and 
direction of this review was strongly informed by a lengthy discussion with one of the 
authors of that review (Hart). 

REVIEW METHODOLOGY 

In its methods, the current review sought to be: systematic (in terms of having clear 
criteria for the inclusion/exclusion of research studies, and a common framework for 
reviewing all included articles); comprehensive (through using a variety of searching 
techniques and checks); and analytical (in terms of critically analysing the strength 
and validity of individual studies’ findings, as well as the evidence base as a whole). 

Selection Criteria 

The key dimensions of the work were clearly defined at the outset (Figure 1). These 
were driven by the focus and rationale outlined above. The focus on school-based 
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environmental education meant that research on any environmental education activity 
undertaken either in school or under the auspices of school (such as a residential field 
course as a school group) was included. What was excluded, then, were 
investigations of activities organised outside of the school context such as summer 
outward bound programmes (e.g. Hattie et al., 1997). The review also excluded 
research on environmental learning amongst university students (e.g. Ballantyne and 
Packer, 1996) and beginning teachers (e.g. Payne, 1997). 



Figure 1 : The Key Dimensions of the Review 



Overall focus: 

Time scale: 

Age range : 
Geographical scope: 

Sources: 



Empirical studies on learning and learners in the context of 
school-based environmental education 

Work published 1993-1999 

Primary and secondary age students (ISCED Levels 1-3) 

International (but, for pragmatic reasons, only articles 
published in English) 

Published articles, books and monographs, and government/ 
international publications 



These exclude: 

♦ Publications that have no empirical component 

♦ Studies of environmental education not undertaken in or through schools 

♦ Studies of teachers, or adult learners or university students 

♦ Research published prior to 1993 or in languages other than English 

♦ Unpublished work such as doctoral and masters theses. 



Search Methods 

Relevant literature was identified from a number of different sources. These 

included: 

♦ Bibliographic data-bases — Searches were undertaken of several educational 
research databases (such as the British Education Index, ERIC, the Australian 
Education Index, and Child Data), as well as more specialist records (SIGLE for 
grey literature, and REGARD for ESRC projects). For all of these the key word 
‘environmental education’ was used, in association with a number of other related 
search terms. These yielded a large number of titles that were then selected 
manually on the basis of abstracts and descriptors. 

♦ Journals/previous reviews - The database searches were supplemented by manual 
searches of the contents of key research journals in the field ( EER , JEE, AJEE, 
CJEE, IRGEE ), and previously published reviews and bibliographies, from which 
articles of potential relevance were identified and copied. 

♦ Researcher networks - A further source of information was researchers currently 
active in the field; requests for information for the review were communicated to 
researchers internationally via the FERN Environmental Education Research 
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Network (UK and Eire), the Environmental Education Research e-mail discussion 
list (international) and the website of the North American Association of 
Environmental Education. 

As well as these initial sources of information, the identification of materials was an 
ongoing process. Reading selected publications led to the identification of further 
references of potential relevance. Indeed this was one way of working towards 
comprehensiveness in the literature search, in that searching continued until no new 
citations arose from the reference lists of included articles. 

The purpose of exploring all of these sources was to identify and collect publications 
that corresponded with the focus of this literature review. This involved searching 
and selecting on the basis of relevance. It is important to emphasise that there was no 
consideration of quality during this initial searching and selection process. (The 
reasoning behind this is explored later in relation to critiquing the research evidence). 
Studies were only excluded if they did not correspond with the stipulated substantive 
focus or failed to match any of the key dimensions outlined above. In the end there 
were 110 publications included in this review, the vast majority of which were journal 
articles, along with a small number of books and research reports. 

Review Processes 

The selection of relevant materials was followed by the process of reviewing 
individual publications. In the interests of ensuring commonality and 
comprehensiveness in this process throughout the project, a review framework was 
devised (Appendix 1). This was adapted from a similar framework used by the author 
during an earlier review of research on young people’s attitudes to education, 
employment and training (Morris et al., 1 999). In the context of the current work, the 
framework was designed to generate information for three purposes: 

♦ cataloguing and reporting - basic descriptive information (such as full publication 
details, geographical location and age of learners), as well as a category descriptor 
(concerning broad substantive focus of a study) were included in order to facilitate 
cataloguing and subsequent analysis and reporting of large numbers of studies; 

♦ evaluation - as well as descriptive information, this framework was also designed 
to generate evaluative information about the depth of detail provided about the 
different aspects of each study (conceptual/theoretical framework, sample, 
methodology, validity measures, methods, main findings, key conclusions, and 
author’s view of implications), and any particular strengths and potential 
weaknesses that were apparent to the reviewer within the work as reported; and 

♦ evidence base analysis - the third purpose of the framework was to enable the 
generation of ideas about (i) the contribution that individual papers made to the 
evidence base (i.e. main findings, key conclusions, author’s view of implications, 
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researcher’s view of implications), and (ii) cases of agreement and disagreement 
between the evidence generated by different papers (i.e. links). 

The process of reading and reviewing articles using the framework began as soon as 
the first references became available. Initially this occurred in an uncoordinated 
manner, in that articles were not examined in any particular sequence, just as they 
arrived. Before long, though, the growing number of relevant publications meant that 
the issue of sorting and categorising became a more pressing one. This marked the 
beginning of a categorising process that continued to evolve throughout the course of 
the review. This involved experimenting with ways of making sense of the evidence 
base through looking for patterns of commonality and differences between the 
substantive foci of individual studies’ findings. 

The emphasis on the substantive foci of their findings, as opposed to their 
methodological approaches, was deliberate in light of the concern with the evidence 
base. This was reflected in the labels that were given to these emerging categories, all 
of which were phrased in terms of ‘Studies that generate evidence about (for 
example) students’ learning outcomes’. In other words, rather than grouping studies 
using a certain methodological approach or focusing on a particular topic, the 
categories in this review constituted an area of the evidence base and the studies that 
contributed to this. Any one study could thus be recorded in several categories if its 
findings were relevant to several areas of the evidence base. 

Once these categories had been established and the majority of the references 
assigned to one or more of them, the articles were reviewed by category. The reason 
for reviewing the articles in this way was to facilitate the development of a more 
synthetic understanding of the findings pertaining to each of the major foci of the 
evidence base. It is important to emphasise that there was flexibility within this 
approach in that individual articles often raised questions about the distinctions 
between categories and necessitated alterations or refinements. This was seen as an 
integral part of the review process. 

Reviewing the research studies within each of the substantive categories involved two 
inter-related processes: evaluation of the strengths and weaknesses of the empirical 
evidence; and the identification of key messages through synthesis of individual 
studies’ findings. The nature and difficulties of these two processes of critiquing the 
research evidence, and synthesising research findings, are explored in the ensuing 
section. 




9 



7 



Mark Rickinson - Paper for presentation at A ERA , Seattle, April 2001 



DIFFICULTIES IN MAKING SENSE OF EVIDENCE IN ENVIRONMENTAL 
EDUCATION RESEARCH 

This section explores in more detail three aspects of the review process which were 
either problematic during the review’s undertaking or have become the source of 
concern since the project’s completion. These focus on: 

♦ critiquing research evidence; 

♦ seeking synthesis; and 

♦ suggesting implications for research users. 

For each of these, an attempt is made to outline, and critically reflect upon, the 
strategy adopted and the problems experienced, and to consider what questions these 
might raise about reviewing evidence in environmental education. 

Critiquing Research Evidence 

It was recognised from the outset that a review focusing on the evidence base would 
need not only to report on recent research findings, but also to evaluate and comment 
upon their quality. This was probably the most challenging part of the review, and the 
aspect of the work that I still feel least sure about. The approach that I adopted was to 
avoid initially the issue of quality in terms of the criteria for selecting studies for the 
review (as described earlier, the criterion used instead was relevance to the 
substantive focus of the review). In other words, unlike many examples of systematic 
research reviews (e.g. Evans and Benefield, 2000), in this review research studies 
were not included or excluded on the basis of their methodological validity. This 
reflected the fact that one purpose of the review was ‘to chart the nature of the 
evidence base on learners and learning’, which necessitated a consideration of all 
(rather than a selection of) the research that had been undertaken on learners and 
learning in environmental education. The onus initially, then, was on generating an 
understanding of what studies had been undertaken, not on analysing the findings 
from a smaller number of studies with particular methodological characteristics. 

Furthermore, I did not see that there were clear-cut quality criteria that I could easily 
adopt for the purpose of investigating environmental education research. Within the 
literature on systematic reviews, for example, ‘the notion of a “hierarchy of 
evidence”, with randomised controlled trials (RCTs) and meta-analyses of RCTs at 
the top, and the opinions of respected authorities, expert communities and descriptive 
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studies at the bottom’ (Davies 2000a: 291-2) seemed unhelpful for making sense of 
the diverse field of environmental research. Meanwhile, within the environmental 
education research literature, while there were research reviews that had undertaken 
methodological critique of quantitative studies (e.g. Leeming et al., 1993), there was 
little agreement over the nature, or appropriateness, of criteria for qualitative inquiries 
(see, for example, contributions to Environmental Education Research 6, 1, Special 
Issue: Qualitative Methods of Inquiry). 

In view of this situation, I chose to proceed in a more inductive manner whereby the 
emphasis was on engaging with the current evidence and seeking to make sense of it, 
rather than working with a set of hard and fast criteria by which to evaluate each 
included study. The way in which this actually took place was through reading 
individual studies and noting points that were felt to be strengths and weaknesses in a 
methodological sense. The emphasis, therefore, was on working in the particular of 
specific studies, and trying to think of ways in which the validity of findings might be 
called into question or the consistency between the findings and the conclusions 
challenged. The aim was to come to an understanding of what kinds of statements 
could, and could not, be made on the basis of individual studies’ findings, and then of 
groups of studies’ findings (the issue of synthesising from different studies is 
considered separately below). This process was informed by: previous reviews and 
methodological papers that had critiqued particular kinds of studies and highlighted 
certain methodological shortcomings and/or strengths; methodological arguments 
within individual studies which in turn raised questions about how other similar 
studies had been undertaken; and my own previous methodological readings, and 
research/literature review undertakings. 



In reviewing individual studies, three principles emerged as useful and important. 

These included: 

♦ Critiquing from within - A conscious effort was made to review pieces of work 
from within the research tradition (or paradigm) that the research had been 
conceived and undertaken. For example, correlational studies were considered in 
terms of positivist research traditions, while qualitative case studies were 
examined from the perspective of interpretivist inquiry. The concern was to 
examine how well the researchers had carried out what they had intended 
according to the paradigm in which they were operating. The aim was not to 
critique articles in terms of the paradigm in which they had chosen to investigate a 
particular topic or question. 
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♦ Sensitivity to context - Care was taken to remain mindful of the situations in 
which research projects take place, and the difficulties that can often hamper such 
work. For example, criticisms that might be leveled at an internationally funded 
research project could be quite unjustified in relation to a small-scale action 
research study by one teacher researcher. Furthermore, research reports differ 
depending upon the circumstances in which they are commissioned and upon their 
intended audiences. For example, methodological details might be less of a 
priority in a publication intended for policy audiences, than one prepared for 
academic readership. 

♦ Distinguishing evidence — Importance was also attached to trying to draw 
distinctions between evidence that seemed to be more reliable or conclusive, and 
evidence that appeared more questionable or preliminary. This involved, for 
example, distinguishing between claims based on empirical findings and those 
based on speculation about empirical findings, highlighting differences between 
statistically significant results and ones based on description of trends, and 
differentiating between survey findings based on very small samples and those 
based on larger representative samples. 



Looking back on this process of critiquing the research evidence during the review, I 
can see that it was characterised by three kinds of difficulty. One problem was simply 
the challenging nature of judging other people’s published research. It is one thing to 
discuss and report upon data in the form of research participants’ (usually 
anonymised) survey responses, interview accounts, and/or observational descriptions, 
it is another to draw upon other researchers’ work as your primary data and 
specifically set out to analyse and evaluate them in a very open and non-anonymous 
manner. Clearly this is a crucial part of the development of any field of research, but 
that does not make it any less challenging either intellectually or personally when it is 
you that has to make the judgements and report them publicly. Issues of quality with 
respect to any research are inevitably highly contentious and complex, and I would 
add that this is particularly so in a field such as environmental education where 
methodological differences are deep-seated and hotly debated. 



A second area of difficulty was the critical appraisal of qualitative evidence. This was 
problematic for two reasons. Firstly, relative to quantitative studies, the reviewing of 
qualitative research has to deal with more diverse genres of inquiry (Hart, 2000; 
Davies, 2000b), and the fact that there are fewer examples of critical analysis of 
qualitative inquiry in previous reviews, and little agreement over criteria of quality 
(see, for example, contributions to Environmental Education Research 6, 1). 
Secondly, there was a marked imbalance within the identified literature between the 
large number of quantitative studies and the much smaller number of qualitative 
inquiries. This presented difficulties in light of the decision not to use a set of clear- 
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cut quality criteria from the outset. This meant that there was a strong dependence on 
comparative analysis between studies in terms of their methodological characteristics, 
rather than an examination of each study in relation to a common set of criteria. More 
detailed critical appraisal was therefore easier where there were larger numbers of 
studies of a similar methodological kind. In this way, the relative shortage of 
qualitative studies within the review meant that the critical evaluation of this type of 
evidence was less detailed and developed. 

A third issue which arose during the conduct of the review was that the critical 
analysis of evidence proved a more complex process than had been originally 
envisaged, in that it operated at different levels. It became clear that strengths and 
weaknesses in evidence could be identified not only at the level of individual studies, 
but also at the level of the evidence base as a whole. Each of these levels involved 
quite different issues and concerns. At the level of individual studies, strengths and 
weaknesses were more in terms of methodological coherence, research designs, 
analytical procedures, validity of claims and so on. At the level of the evidence base, 
meanwhile, issues such as the diversity of evidence types, the nature and variety of 
substantive foci, and the extent of interconnections between types of evidence and 
types of foci, were more important. This meant that it became necessary to think 
carefully about the level at which different kinds of strengths and weaknesses were 
relevant, and how these different levels might inter-relate, overlap or even conflict. 
My feeling is that the idea of different levels of critique is an important one for the 
future development of evidence-focused reviews, particularly as much of the 
discussion about systematic reviews of research evidence seems to have focused more 
on the question of judging individual studies, and less on the issue of what might be 
seen as judging the quality of research fields. 

Stemming from these three difficulties with critiquing research evidence, I would 
suggest that future reviews of environmental education research evidence may well 
benefit from: 
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being undertaken by a number of researchers working collaboratively, rather 
than by single individuals working in relative isolation - This recognises the 
personally and intellectually challenging nature of reviewing, critiquing and 
synthesising research. It also responds to the crucial need for in-depth discussion 
and debate about the evidence during the process of a review, which would 
arguably be facilitated by a collaborative team approach. Indeed, Hart and Nolan 
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(1999:3) in their review, report how they ‘engaged in reflective discourse to create 
a collaborative interpretation and a critique of the research’. Such a way of 
working might be informed by procedures and processes developed in connection 
with systematic reviews of research in the health sector (Sheldon and Chalmers, 
1994) and, more recently, in education (Evans and Benefield, 2000) with their 
inclusion of advisory groups of experts as part of the review process. 

♦ a deeper engagement with literature and debates on criteria of 
methodological quality before , as well as during, the review process - This 
would, of course, need to include consideration of ideas both within and beyond 
the environmental education field. The purpose, though, would be to generate, in 
an involved and reflexive manner, indicators of quality for different genres of 
evidence that would be expected to feature in the review. While developed at the 
outset, it would also be important, in my view, for these indicators or criteria to be 
open to question during the review process. Of particular importance, I would 
suggest, might be situations where the indicators did not seem relevant or 
appropriate to a particular study or group of studies and therefore in need of 
amendment, or whole-scale revision. This process of amendment or revision, 
though, would be reported as part of the review, with the hope that different 
reviews would provide not only new framings of the field and syntheses of 
findings, but also new ways of conceiving and evaluating research evidence. 

♦ greater consideration being given to the notion that research evidence can be 
conceptualised and evaluated at different levels - Two levels that have emerged 
from this review are the level of individual studies and the level of the evidence 
base as a whole. There may well be other possible levels, and indeed, more 
sophisticated ways of conceptualising research fields and evidence bases that will 
come from further exploration. My sense is that more attention could usefully be 
given to the idea of quality of research fields, as well of individual research 
studies. One challenge is of developing language and metaphors with which to 
talk about evidence bases in this sense. 

Seeking Synthesis 

Another central aim of the review was to ‘identify key messages emerging from the 
evidence base for each of the major foci within the area’. In other words, there was a 
desire to synthesise the findings of different studies on similar topics in terms of key 
messages or ‘overarching summarising statements’, which not only included all of the 
available evidence on a particular topic, but also reflected (in their phrasing) the 
specificity and strength of the findings upon which they were based. 




The interest in synthesis stemmed from a desire to make sense of evidence about 
learners and learning in environmental education in a way that might be useful to 
research users, as well as researchers. As outlined earlier, the review was informed 
by the view that previous reviews in the field had focused on methodological matters 
more than on research findings. Their products were therefore arguably more relevant 



14 



12 



Mark Rickinson - Paper for presentation at A ERA, Seattle, April 2001 



and useful to researchers, than to research users. In line with the argument that ‘the 
main channel of communication between researchers and lay people ought to be 
reviews of whole fields of research, rather than reports of single studies’ (Foster and 
Hammersley, 1998:610, Davies, 2000b:366), value was seen in seeking ways of 
summarising the findings from groups of studies focused on particular aspects of 
learners or learning in environmental education. 

The way I undertook this was in terms of exploring the areas of commonality and 
difference between studies’ findings, and trying to generate overarching summarising 
statements which encompassed and expressed these as key messages. This kind of 
approach, I see now, was similar to the way in which I approached the task of 
critiquing the research. That is, I did not use a specific named a priori method from 
the outset. The main reason for this was that the methods of research synthesis which 
featured in the literature either required the selection of particular kinds of ‘high 
quality’ studies (such as ‘best evidence synthesis’), or seemed strongly orientated 
towards quantitative research with little relevance to other kinds of inquiry (such as 
‘vote counting methods’ or ‘meta analysis’) (for details on these approaches see, for 
example, Davies, 2000b). Given these kinds of available techniques, none of which 
were felt to be adequate for the task of investigating the nature and quality of 
environmental education learning research, I opted instead for a more inclusive and 
inductive approach to synthesis. This involved generating overarching summarising 
statements through reading, rereading and trying to synthesise the findings reported in 
each of the studies for a particular substantive focus. In seeking to identify key 
messages across studies, attention was paid to trying to include all of the evidence 
available on a particular topic, as well as phrasing the summarising statement in a way 
that reflected the nature and strength of the findings upon which it was based. 

Looking back, this process of synthesis was conceived and undertaken in a relatively 
unproblematic manner. Unlike the recurring challenges and concern associated with 
the critiquing of the evidence, the generation of key messages was not found to be an 
inherently difficult process during the review. That is not to say that there were not 
challenges in pulling together the large amount and (at times) wide variety of 
empirical findings for particular topics, or with developing the wording of 
summarising statements so as to reflect subtleties in a succinct manner. These sorts of 
challenges, though, were experienced more as practical ones, rather than as being 
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threatening in a conceptual or theoretical way. It is only now that I am beginning to 
question what this process was actually about, what assumptions it made, what its 
limits might have been, and how it might have been more carefully theorised and 
operationalised. 

There are two particular concerns that have emerged through reflection since the 
completion of the review. The first of these is the concern that synthesis can result in 
an oversimplification of research evidence, whereby the impression is given of 
greater clarity within the research evidence than is, in fact, warranted. This is a point 
raised by Gough (1999) in relation to a review of significant life experiences research 
(Chawla, 1998a), which made a statement about two different studies yielding 
‘markedly similar results’. Gough (1999:413-4) criticises this statement for falsely: 

‘inviting the reader to accept that the different methods and questions “reveal” 
parts of the same “objective picture” [...when in fact ...] The similarity 
between [the two studies’ findings] is not surprising given that they were both 
working in the same research tradition, using the same disciplined procedures 
for data production and interpretation’. 

This notion of attaching too much significance to fairly inevitable similarities 
underlines the need for synthesis to pay careful attention not only to similarities and 
differences in the findings of studies, but also to similarities and differences in the 
methodological characteristics of studies. Another potential source of over- 
simplification comes not so much from the overplaying of similarities, but from the 
overlooking of differences. The risk here is that an attempt to synthesise can easily 
become overly concerned with identifying similarities between studies, rather than 
with highlighting their differences. 

A second concern about synthesis is whether it is a process based on a positivist logic, 
and therefore better suited to dealing with quantitative findings, rather than qualitative 
ones. This is the argument put forward by Noblit and Hare (1988:12), who see most 
approaches to knowledge synthesis (such as meta- analysis) as ‘quantitative 
approaches [which] require a determination of a basic comparability between 
phenomena so that data can be aggregated for the analysis’ (ibid.: 17). This, in their 
view, is unhelpful for the purpose of dealing with interetivistic research because it 
ignores the ‘meaning in context’ and the ethnographic uniqueness that is so central to 
ethnographic and qualitative inquiry. On this basis, they propose the alternative 
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concept of ‘meta-ethnography’ which ‘seeks to go beyond single accounts’ by 
revealing ‘the analogies between the accounts’ through the ‘translation of qualitative 
studies into each other’ (rather than the development of ‘overarching generalisations’) 
(ibid.: 13 and 25). 

Unfortunately, I was not aware of this approach at the time of undertaking the review. 
Thinking about it in relation to the review, though, not only highlights the importance 
of synthesis in reviewing research, but also raises a number of crucial questions about 
what is involved in synthesis and what kinds of synthesis are useful for which types of 
evidence. In particular, questions emerge about: 

(i) the subject matter of synthesis - exactly what aspects of individual research 
studies are we trying to synthesise, and can/should this vary for different kinds 
of research? 

(ii) different types of synthesis — if meta-analysis is positivistic, and meta- 
ethnography is interpretivistic, then what other kinds of synthesis are possible 
and what might they look like? 

(iii) synthesis between paradigms - if synthesis itself is paradigmatic, then is 
synthesis across different paradigms/genres of research possible? 

(iv) the limits of synthesis - are there situations where more is lost through 
synthesis than is gained, and, if so, what alternative processes might be more 
appropriate? 

Overall, it would seem that there is a need to open up the notion of synthesis for 
closer scrutiny in relation to different kinds of research evidence and different kinds 
of reviews for different kinds of audiences. What is important is that synthesis does 
not necessarily have to mean statistical meta-analysis; alternatives notions such as 
meta-ethnography are emerging and reviewers of environmental education research 
would seem well placed to draw upon, and contribute to, the development of such 
approaches. 

Suggesting Implications for Research Users 

In line with its focus on the evidence and what it might tell us, the review sought not 
only identify key messages, but also to highlight issues and challenges arising from 
these for research users, as well as researchers. This task of suggesting implications 
for research users proved a further challenging aspect of the review. Difficulties, for 
example, were experienced with: 
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♦ the wide variety of potential user groups in environmental education, and trying to 
think about which kinds of users might be interested in the evidence discussed in 
the review and the ways in they might make use of it; 

♦ the recognition that research evidence will rarely translate easily into simple 
ingredients for developing environmental education practice or policy, particularly 
as ‘factual information cannot, in itself, tell us what should be done’ (Foster and 
Hammersley, 1998:621); 

♦ trying to work out which research evidence should be mentioned in what ways, in 
how much detail, and with what qualifiers, in relation to implications; and 

♦ the danger that ‘practical recommendations [can] effectively close down 
discussion of those issues’ with possible negative consequences for the 
development of educational provision and reflective practice (Foster and 
Hammersley, 1998:624). 

On reflection, I see that my identification of implications for research users was 
characterised by two main strategies. Firstly, I tried to raise issues or highlight 
findings for consideration by research users, rather than making specific 
recommendations about how practice ought to change in light of research findings. 
For example, a number of characteristics of learners emerging from the research 
evidence (such as the idea that students’ curricular and pedagogical preferences can 
differ) were highlighted for consideration in relation to the question of ‘how [they] 
might be taken into account in teaching and learning practices within environmental 
education’. Thus, I did not attempt to actually stipulate possible ways in which 
practice might need to change in order to take account of such learner characteristics. 

Secondly, I tried to emphasise the way in which research findings need to be seen in 

relation to the assumptions that the research has made about environmental teaching 

or learning, and the nature, aims and context of the particular programme that was 

being researched. For example, in reporting certain aspects of environmental science 

teaching that research on students’ ideas about global environmental issues had 

identified as detrimental to students’ understanding of such issues, it was stressed how 

the studies ‘each make assumptions about the nature of environmental teaching which 

may or may not be shared by individual research users coming to consider their 

possible implications’. This bears some similarity to Foster and Hammersley’s 

(1998:623) argument that if evaluations or prescriptions are to be drawn then: (i) these 

are better expressed in ‘conditional’ terms, that is ‘given commitment to this goal, 

then policy or practice meets or falls short of what is required in the following 

respects; or, given this goal and these particular value concerns about means, the 

following course of action would be desirable’; and (ii) they need to fully explicate 
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the criteria by which a policy or practice is seen to be succeeding or failing (for 
evaluations), and/or the assumptions being made about goals and means (for 
prescriptions). That said, I have no doubt that I could have gone further in stipulating 
the kinds of conditions and assumptions that underpinned the findings which featured 
in the section on implications for research users. 



Overall, I am sure that the review did no more than scratch the surface of the 
challenges inherent in undertaking reviews for the purpose of communicating with 
research users. One thing it has made clear, though, is that reviews are not just about 
an understanding of research, but also an understanding of practice, and most 
crucially, the relationship between research and practice. This calls into question the 
idea of reviews being undertaken solely by researchers. The review I undertook 
represents, what Bassey (2000) would term, an ‘academic review’. Bassey draws a 
distinction between an academic review, ‘a critical and analytical account of the state 
of public knowledge of the topic’ aimed primarily at other researchers, and a user 
review: ‘a form of professional paper which is devised and written by researchers and 
users working together’ with the aim of critically informing the thinking of a 
particular policy maker or practitioner audience (ibid.:25). The latter, he explains, 
would arise from an academic review but would be much briefer and more readable in 
style and, most importantly, would focus on specific issues of relevance to particular 
groups of users. It would seem that reviews of this kind are currently rare in 
environmental education, but could, in fact, represent an important and promising 
opportunity for the field, its practitioners and its researchers. I would add, though, 
that their undertaking needs to be informed by ideas and debate about: 



♦ the relations and differences between the worlds of research and those of 
practice - In a paper entitled ‘Why research into practice does not go’ 
Hammersley (1998), for example, proposes a number of limitations on the ability 
of research to provide useful insights for practice. These include: the fallibility of 
research-based knowledge, the relatively slow rate of research knowledge 
generation, its tendency to focus on single issues, its generalised character, and its 
inability in itself to justify practical evaluations and recommendations. 
Furthermore, he argues that, practitioners act upon much more than factual 
knowledge, and ‘research will only have an impact if it finds a place in a 
motivational context which encourages practitioners to use it.’ (ibid.: 16). 

♦ the possibility that research knowledge can have negative as well as positive 
consequences for practice - Another point made by Hammersley (1998:17-21) is 
that the production and dissemination of research knowledge can be detrimental to 
practical decisions and outcomes where research knowledge is: false and so leads 
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practitioners astray, sound but is used for bad purposes, too plentiful and so 
confusing, too complexifying (as opposed to simplifying) and so demotivating or 
disorientating, inappropriate for the local context, and used to legitimate bad 
courses of action and vice versa. 

♦ the multi-faceted and heterogeneous nature of research users — Foster and 
Hammersley (1998:613) warn that ‘it is important to remember that there is not 
one single, well-defined, and internally homogenous professional audience, but 
rather multiple, overlapping publics whose boundaries and characteristics are ill- 
defined’. 

♦ the potential impact of ‘researchers as sideline audiences’ — Foster and 
Hammersley (1998:612) also highlight how ‘even if the intended audience is non- 
researchers, the reviewer will nevertheless be aware that fellow researchers are 
also likely to read the review’ and this can ‘distract the reviewer from providing 
what would be most helpful for policy-makers or practitioners’. 

These kinds of ideas may well be important in shaping responses to calls for 
environmental education to become more evidence-based, as seen in the earlier 
quotes from Palmer (1999) and Reid (2001). 



CONCLUSION 

At its heart, this paper is about the complexity of reviewing research evidence in 
environmental education. Set against arguments for environmental education to 
become more evidence-based (e.g. Palmer, 1999; Reid, 2001), it therefore represents a 
call for a greater appreciation of, and more critical discussion about, the challenges 
inherent in making sense of evidence in the field through reviews of research. 



Bassey (2000:24) has argued that the undertaking of research reviews ‘deserves the 
same level of academic esteem as the conduct of empirical research’. I would echo 
this, but argue that it is not only the same level of ‘academic esteem’ that is needed, 
but more importantly the same level of methodological reflexiveness and deliberation. 
The issues discussed in this paper have made clear that reviews, like all research 
endeavours, are intricately tied up with a whole host of methodological challenges. In 
particular, they are shaped by varying conceptions of research and evidence, differing 
notions of synthesis, ideas about the relationship between research and practice, and 
issues of audience and representation. 



ERIC 



There are, though, many kinds of research reviews, and many ways of making sense 
of, and synthesising, research (Davies, 2000b). I would argue that the methodological 
diversity of the current environmental education research field is not yet matched by a 
similar diversity of approaches for reviewing and making sense of this research. I see 
an important need, therefore, not only for a greater number and variety of research 
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reviews in the field, but also for these to have much stronger connections to the field’s 
tradition of methodological discourse and debate. 
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Appendix 1: The Review Framework 



Reference: 



Category Descriptor: the emerging category(ies) of relevance Date of review: 
Country of Origin: country(ies) where the work was undertaken Age of learners: 



Research 

aims 


a summary of the aims of the research study as reported by the 
researcher in their paper 


Project details 


name and funding details of any broader research project (if 
mentioned) 


Conceptual/ 

theoretical 

approach 


summary of the key conceptual and/or theoretical assumptions 
that underpin the work reported (but only in so far are these are 
explicated and acknowledged by the author) 


Sample 


details of sample sizes, sample characteristics, and selection 
procedures and rationale 


Methodology 


the broader epistemological and theoretical framework that 
surround and underpin the methods of the study (but again 
only in so far are these are explicated and acknowledged by the 
author) 


Validity 

measures 


any measures aimed at ensuring validity or reliability 
(howsoever conceived) that are reported by the author 
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Methods 

(data 

collection and 
analysis) 


summarised details of the reported procedures of data 
collection, and of data analysis 


Main findings 


summary of the study’s main findings as reported by the 
researcher 


Key 

conclusions 


summary of the main conclusions drawn from the study’s 
findings by the researcher 


Author’s view 
of what 
findings tell 
us 


summary of the key implications and lessons that the 
researcher draws from the study 


Reviewer’s 
view of what 
findings tell 
us 


reviewer’s view of the key implications and lessons emerging 
from the study 


Strengths and 
weaknesses 


aspects of the study that the reviewer perceives as particularly 
valuable or potentially problematic/limited 


Links 


brief notes about 

(i) points of commonality or divergence between this and 
other studies in the review e.g. similar or very different 
findings on a similar topic, methodological links or 
conflicts etc. 

(ii) links with other categories within the review, or 
examples of significant differences between papers 
within the same category (leading to sub-categories) 
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